AFRL-APOSR-VA-IR- 2015-0291 


Dynamic  Data  Driven  Operator EiiorEai1yV\i^ming  System 


AlokChaturvedi 
PURDUE  UNIVERSnV 


Oa/ 13/ 2015 
Final  Repoit 


DISIRIBUnON  A:  Distribution  approved  for  public  lelease. 


Air  Force  Research  Laboratory 
AF  Office  Of  Scientific  Research  (APOSR)/  RIA2 
Arlington,  Viiginia  22203 
Air  Force  Materiei  Command 


FORM  SF  298 


Page  1  of 2 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704-0188 

The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing 
data  sources,  gathering  and  maintaining  the  data  needed,  and  compieting  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or 
any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  the  burden,  to  Department  of  Defense,  Executive  Services,  Directorate  (0704-0188). 
Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information 
if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ORGANIZATION. 

1.  REPORT  DATE  fDD-MM-yyyyj  2.  REPORT  TYPE  3.  DATES  COVERED  (From  -  Toj 

23-09-2015  Final  Performance  15-04-2014  to  14-04-2015 

4.  TITLE  AND  SUBTITLE 

Dynamic  Data  Driven  Operator  Error  Early  Warning  System 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

FA9550- 14- 1-0058 

5c.  PROGRAM  ELEMENT  NUMBER 

61102F 

6.  AUTHOR(S) 

Alok  Chaturvedi 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S]  AND  ADDRESS(ES) 

PURDUE  UNIVERSITY 

401  SOUTH  GRANTS! 

WEST  LAFAYETTE,  IN  47907-2024  US 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES] 

AF  Office  of  Scientific  Research 

875  N.  Randolph  St.  Room  31 12 

Arlington,  VA  22203 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

AFRL/AFOSR  RTA2 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S] 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

A  DISTRIBUTION  UNLIMITED:  PB  Public  Release 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

Mitigating  human  errors  is  a  priority  in  the  design  of  complex  systems,  especially  through  the  use  of  body 
area  networks.  This  paper  describes  early  developments  of  a  dynamic  data  driven  platform  to  predict 
operator  error  and  trigger  appropriate  intervention  before  the  error  happens.  Using  a  two-stage  process, 
data  was  collected  using  several  sensors  (e.g.  electroencephalography,  pupil  dilation  measures,  and  skin 
conductance)  during  on  established  protocol  -  the  Stroop  test.  The  experimental  design  began  with  a 
relaxation  period,  40  questions  (congruent,  then  incongruent)  without  a  timer,  a  rest  period  followed  by 
another  two  rounds  of  questions,  but  under  increased  time  pressure.  Measures  such  as  workload  and 
engagement  showed  responses  consistent  with  what  is  known  for  Stroop  tests.  Dynamic  system  analysis 
methods  were  then  used  to  analyze  the  raw  data  through  principal  components  analysis  and  least  squares 
complex 

exponential  method.  The  results  show  that  this  algorithm  has  the  potential  to  capture  mental  states  in  a 
mathematical  fashion,  thus  enabling  the  possibility  of  prediction. 

15.  SUBJECT  TERMS 

human  performance  sensor  controll  decision-making 

U.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF  RESPONSIBLE  PERSON 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

ABSTRACT 

OF 

Alok  Chaturvedi 

Standard  Form  298  (Rev.  8/98! 

Prescribed  by  ANSI  Std.  Z39.U 

DISTRIBUTION  A:  Distribution  approved  for  public  release. 


https://livelink.ebs. afrl.afmiFlivelink/llisapi. dll 


9/28/2015 


FORM  SF  298 


Page  2  of  2 


Unclassified 

Unclassified 

Unclassified 

PAGES 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

UU 

765-494-9048 

standard  Form  298  (Rev.  8/98) 
Prescribed  by  ANSI  Std .  Z39 . 1 8 


DISTRIBUTION  A:  Distribution  approved  for  public  release. 


https://livelink.ebs. afrl.afmiFlivelink/llisapi. dll 


9/28/2015 


This  space  is  reserved  for  the  Procedia  header,  do  not  use  it 


Dynamic  Data  Driven  Approach  for  Modeling  Hnman 

Error 

Wan-Lin  Hu^,  Janette  J.  Meyer^,  Zhaosen  Wang^,  Tahira  Reid^,  Douglas  E. 
Adams^,  Sunil  Prabnakar^,  and  Alok  R.  Cliaturvedi"^ 

^  School  of  Mechanical  Engineering 
Purdue  University,  West  Lafayette,  Indiana,  U.S.A 
hul88@purdue . edu 

^  Laboratory  of  Systems  Integrity  and  Reliability  (LASIR) 

Vanderbilt  University,  Nashville,  Tennessee,  U.S.A. 
j  anette . j . meyerOvanderbilt . edu 
®  Department  of  Computer  Sciences 
Purdue  University,  West  Lafayette,  Indiana,  U.S.A. 

^  Krannert  School  of  Management 
Purdue  University,  West  Lafayette,  Indiana,  U.S.A. 


Abstract 

Mitigating  human  errors  is  a  priority  in  the  design  of  complex  systems,  especially  through  the 
use  of  body  area  networks.  This  paper  describes  early  developments  of  a  dynamic  data  driven 
platform  to  predict  operator  error  and  trigger  appropriate  intervention  before  the  error  happens. 
Using  a  two-stage  process,  data  was  collected  using  several  sensors  (e.g.  electroencephalography, 
pupil  dilation  measures,  and  skin  conductance)  during  an  established  protocol  -  the  Stroop 
test.  The  experimental  design  began  with  a  relaxation  period,  40  questions  (congruent,  then 
incongruent)  without  a  timer,  a  rest  period  followed  by  another  two  rounds  of  questions,  but 
under  increased  time  pressure.  Measures  such  as  workload  and  engagement  showed  responses 
consistent  with  what  is  known  for  Stroop  tests.  Dynamic  system  analysis  methods  were  then 
used  to  analyze  the  raw  data  through  principal  components  analysis  and  least  squares  complex 
exponential  method.  The  results  show  that  this  algorithm  has  the  potential  to  capture  mental 
states  in  a  mathematical  fashion,  thus  enabling  the  possibility  of  prediction. 

Keywords:  bio-sensors,  dynamic  data-driven  application  systems  (DDDAS),  least  squares  complex 
exponential  (LSCE),  error  detection 


1  Introduction 

Human  factor  studies  have  shown  that  nearly  80%  of  the  root  causes  contributing  to  major 
accidents  affecting  safety,  the  environment,  and/or  economics  can  be  attributed  to  human 
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operator  error  [18].  Causes  for  the  human  operator  error  are  many.  In  a  report  for  the  TNO 
Institute  for  Perception,  Raaijmakers  [19]  discussed  the  impact  of  mental  and  physical  stress 
on  human  performance  in  decision-making  situations,  specifically  for  time  critical  decisions. 
Raaijmakers  argued,  “  . . .  effects  of  various  stressors  (fatigue,  sleep  loss,  time  pressure,  anxiety, 
and  cognitive  strain)  are  shown  to  be  task-dependent.  At  lower  levels,  fatigue  and  sleep  loss 
seem  to  be  the  most  important  stressors;  at  higher  levels,  the  largest  effects  are  expected  to  be 
time  pressure  and  cognitive  strain.”  (p.  1). 

This  paper  describes  preliminary  developments  of  a  dynamic  data  driven  platform  to  predict 
operator  error  and  trigger  appropriate  intervention  before  the  error  happens.  Taking  advantage 
of  the  advances  in  commercial  grade  psychophysiological  sensors  such  as  electroencephalogra- 
phy(EEG),  electrocardiogram(ECG),  galvanic  skin  response  (GSR),  and  eye-tracking  systems, 
a  human  operator’s  mental  and  physical  states  before,  during,  and  after  each  cognitive  activity 
is  captured  in  an  android  device  which  doubles  as  the  server  for  a  wireless  body  area  network 
(WBAN).  Thus  the  operator  becomes  an  integral  part  of  the  system  through  a  brain  computer 
interface(BCI).  WBAN  server  assimilates  and  fuses  data  from  multiple  sensors  and  streams  it 
to  a  stress-intensity  simulation  model,  which,  in  turn  may  trigger  appropriate  intervention. 

The  rest  of  the  paper  is  organized  as  follows:  Section  2  describes  the  methodology.  Section  3 
presents  the  preliminary  results.  Section  4  discusses  approaches  to  raw  data  analysis;  Section  5 
presents  the  results  and  analysis;  and  Section  6  concludes  the  paper. 


2  Background  Literature 


By  virtue  of  being  human,  it  is  understood  that  people  will  make  errors  [21].  The  field  of  aviation 
has  been  the  most  prominent  in  developing  methods  for  evaluating  and  preventing  human  error 
through  processes  and  protocols  [13,  17].  In  recent  years,  a  number  of  interventions  have  been 
developed  to  help  mitigate  human  errors.  In  the  automotive  industry,  new  technologies  have 
been  introduced  that  automatically  slow  vehicles  down  prior  to  rear-ending  a  car  or  potentially 
hitting  a  pedestrian  [6].  Such  methods  work  through  technologies  embedded  in  physical  systems. 

Psychophysiological  measurements  provide  a  means  for  gathering  data  directly  from  people. 
These  data  are  physiological  measurements  that  correspond  to  psychological  states.  EEG  is  a 
common  sensor  that  has  been  used  in  studies  to  predict  future  and  present  cognitive  perfor¬ 
mance  [20]  and  to  develop  a  driver  drowsiness  monitoring  system  [12].  In  [9]  a  body- mounted 
system  is  developed  to  prioritize  incoming  data  streams  based  on  the  users  workload,  which  is 
measured  using  EEG  and  EGG  signals. 

Recent  advances  in  wireless  communication  and  wearable  devices  have  led  to  the  develop¬ 
ment  of  wireless  body-area  networked  systems.  Eletcher  et  al.  developed  a  wearable  sensor 
platform  for  integrating  data  from  skin  conductance  and  heart  rate  to  monitor  emotion  and 
behavior  for  clinical  use.  Cognitive  Behavioral  Therapy  could  be  given  automatically  based  on 
bio-signals  [11].  Choi  et  al.  established  a  wearable  mental  stress  monitoring  system  based  on 
heart  rate,  respiration,  electromyography  (EMG),  and  electrodermal  activity  (EDA)  [5].  Kumar 
et  al.  further  introduced  a  mobile  health  system  structure  to  recognize  behaviors  and  deliver 
timely  interventions  [15].  Although  several  studies  have  proposed  wearable  biosensor  systems 
to  track  stress  or  behavior,  the  method  for  preventing  errors  based  on  these  measures  is  not 
well-established. 
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(a) 


(b) 


Figure  1:  Screen  shots  of  the  computer-based  Stroop  test,  (a)  A  congruent  question  with  a 
3-second  timer,  (b)  An  incongruent  question.  The  participant  has  ran  out  of  time. 


3  Methodology 

3.1  Subjects 

The  study  protocol  was  approved  by  the  Institutional  Review  Board  on  human  investigation. 
A  total  of  10  adults  volunteered  to  join  the  study  after  passing  the  prescreening  and  providing 
a  written  consent  form.  The  first  5  participants  were  in  an  initial  characterization  tests  to  help 
us  refine  the  experimental  procedure.  We  only  report  the  data  from  the  second  5  participants 
(2  women  and  3  men)  who  joined  the  formal  study. 

3.2  Computer-Based  Stroop  Test 

In  order  to  better  record  response  time  and  synchronize  it  with  biosignals,  a  computer-based 
Stroop  test  developed  on  Qt  5.3  (Qt  Project  Housing,  Norway)  was  used  in  the  study  (Figure  1). 
To  make  the  question  colors  easy  to  distinguish,  we  only  include  primary  colors:  red,  blue,  and 
green,  which  is  consisted  with  established  Stroop  test  protocols  [7] .  Moreover,  participants  used 
arrow  keys  (left,  down,  and  right)  to  answer  questions.  It  was  intuitive,  so  participants  could 
answer  as  quickly  as  possible  when  they  figured  out  the  answer.  The  result  of  each  question 
showed  on  the  screen  for  0.5  second.  If  the  participant  did  not  answer  in  time,  it  showed 
’’timeout”  and  was  recorded  as  a  wrong  answer  on  the  question. 

3.3  Equipment 

The  main  approach  of  this  research  is  to  develop  a  body  area  network  to  capture  signals  from  bio¬ 
sensors.  In  order  to  obtain  holistic  psychophysiological  responses  of  participants  during  decision 
making  (and  various  kinds  of  cognitive  activities),  3  types  of  sensors  were  used  in  the  proposed 
experiment.  A  B- Alert  XIO  EEG  headset  system  (Advanced  Brain  Monitoring,  Inc.,  USA),  a 
Shimmer3  GSR-I-  Unit  (Shimmer,  USA),  and  a  Tobii  X60  eye  tracker  (Tobii  Technology,  USA) 
were  utilized  to  collect  brain  waves,  galvanic  skin  response,  and  pupil  dilation,  respectively.  All 
data  collection  was  performed  and  synchronized  through  iMotions  Attention  Tool  (iMotions, 
Inc.,  USA)  on  Windows  7  operation  system. 

B- Alert  XIO  is  a  9-channel  EEG  system,  which  records  brain  waves  from  mid-line  and  lateral 
sites  on  the  scalp  including  Fz,  Cz,  POz,  F3,  C3,  P3,  F4,  C4,  P4  (Figure  2).  In  addition  to  raw 
waves  measured  from  the  above  9  sites  in  256  Hz  sampling  rate,  B- Alert  XIO  generates  4  kinds 
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Figure  2:  Locations  and  labels  of  data  channels  acquired  using  the  B- Alert  system  [1]. 


B-AlcrtXIO 
EEG  headset 


ShiiTimer3  GSR+  unit 


Computer-based 
Stroop  test 

Tobii  X60  eye  tracker 


Figure  3:  A  participant  wears  the  GSR  sensor  on  the  left  (nondominant)  wrist  and  the  EEG 
head  set,  and  uses  the  keyboard  to  do  the  computer-based  Stroop  test.  The  eye-tracking  system 
is  mounted  under  the  screen  monitor  for  tracking  the  participants  pupils. 


of  metrics  in  an  epoch  of  one  second  for  representing  cognitive  states,  which  are  engagement, 
workload,  distraction,  and  drowsy. 

Participants  skin  resistances  between  2  electrodes  attached  to  the  index  and  the  middle 
fingers  of  the  nondominant  hand  were  monitored  by  the  Shimmer  3  GSR-I-  Unit  in  52  Hz 
sampling  frequency.  Skin  resistance  fluctuates  in  response  to  external  or  internal  stimuli  [10]. 
We  introduced  GSR  to  the  body  area  network  to  detect  mental  stress  response. 

The  proposed  experiment  is  a  computer-based  test,  so  a  Tobii  X60  eye  tracker  was  mounted 
on  the  screen  to  track  how  participants  pupil  dilation  responded  to  stimuli  on  the  screen  monitor. 
Eigure  3  shows  the  experimental  setup  of  the  EEG  headset,  the  GSR  sensor,  and  the  eye-tracking 
system. 

Our  ultimate  goal  is  to  build  a  cost-efficient,  compact,  and  portable  error  early  warning 
system,  and  NeuroSky  MindWave  Mobile  (NeuroSky,  Inc.,  USA)  will  be  a  possible  solution. 
NeuroSky  is  a  non-invasive  EEG  that  connects  the  user  to  iOS  and  Android  platforms,  and 
transfers  all  signal  information  through  Bluetooth  as  opposed  to  radio.  It  can  detect  brainwaves 
ranged  from  0.1  Hz  to  30  Hz,  and  divide  them  into  delta,  theta,  low/  high  alpha,  low/  high 

4 


DISTRIBUTION  A:  Distribution  approved  for  public  release. 


Dynamic  Data  Driven  Approach  for  Modeling  Human  Error 


Hu,  et  al. 


Figure  4:  Example  UI  of  HelloEEG.  Part  1  represents  the  signal  strength  Part  2  is  the  sampled 
data  of  8  wave  ranges.  Part  3  is  the  timestamp.  Part  4  is  the  attention  and  meditation  level 
based  on  NeuroSky  algorithm. 


beta,  and  low/high  gamma  parts.  An  app  called  HelloEEG  was  developed  to  dump  out  the 
data  from  Neurosky  as  in  Figure  4.  All  data  are  sampled  once  per  second.  The  attention  and 
meditation  level  are  calculated  based  on  the  8  wave  bands.  Attention  indicates  the  user  level 
of  focus  and  meditation  indicates  the  user  level  of  relaxation. 

3.4  Procedure 

Participants  were  briefed  with  the  experimental  procedure  and  signed  the  consent  form  at  the 
beginning  of  the  experiment.  After  setting  up  the  EEG  and  GSR  sensors,  participants  were 
asked  to  follow  instructions  to  perform  a  series  of  tasks  for  establishing  the  EEG  baseline. 
The  personalized  baseline  was  used  to  generate  and  normalize  cognitive  metrics:  workload, 
engagement,  distraction,  and  drowsy  measures.  Then  participants  were  asked  to  follow  a  dot 
on  the  screen  to  calibrate  the  eye  tracker.  The  preparation  process  took  approximate  50  minutes. 

The  experimental  procedure  is  shown  in  Figure  5.  After  finishing  calibrations,  participants 
started  to  do  the  experiment  with  a  3-minute  baselining  session.  They  were  rested  and  listened 
to  the  music  Bachs  Harpsichord  Concerto  No.  5  in  F  Minor  BWV  1056,  which  meets  the 
criterion  for  being  of  a  relaxing  nature  [4] ,  and  their  biosignals  recorded  from  this  session  were 
seen  as  a  relaxed  state.  Followed  by  the  baseline  session  was  20  congruent  questions  without 
timer.  Within  these  20  questions,  text  and  color  were  the  same  and  participants  were  able 
to  answer  questions  at  their  own  pace.  Next  was  20  incongruent  questions  without  timer. 
Participants  were  then  given  a  2-minute  rest  session,  and  did  20  congruent  and  20  incongruent 
questions  with  3-second  timers.  This  was  followed  by  another  2-minute  rest  session,  and  did 
one  more  set  of  20  congruent  and  20  incongruent  questions,  but  with  1.5-second  timers.  Stroop 
effect  interferes  with  the  reaction  time  of  a  task,  thus  we  expected  participants  were  going  to 
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y  No  Timer 

'  3-second  Timer  ^ 

y  1. 5-second  Timer^ 

/■  \ 

BliNclinc 
(3  mins) 

20  C. 
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20  IC. 

Questions 

V  > 

Rest 
(2  mins) 

20  C. 
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J 

20  IC. 
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/  \ 

Suncy 
(3  mins) 

Figure  5:  The  proposed  experimental  procedure.  C:  congruent;  IC:  incongruent. 


make  more  errors  in  incongruent  questions  with  time  pressure. 

Finally,  participants  answered  a  short  survey  about  their  previous  experience  in  Stroop  test 
and  their  feelings  of  the  timers. 


4  Preliminary  Results 

For  obtaining  an  overview  of  each  cognitive  state,  the  average  values  of  each  metric  over  the 
time  spent  on  each  session  were  calculated.  Therefore,  there  were  10  data  points  of  each  metric 
for  each  participant.  Figure  6  and  Figure  7  show  the  level  of  engagement  and  workload  of 
all  participants  through  the  test.  Although  there  were  differences  between  participants,  they 
tended  to  show  lower  engagement  during  rest  sessions  including  the  baseline  session,  and  had 
higher  engagement  when  getting  familiar  with  the  test  (i.e.,  the  first  20  congruent  questions) 
and  under  considerable  time  pressure  (i.e.,  the  last  20  incongruent  questions  with  1.5-second 
timers).  S003  had  a  relatively  low  engagement  level  in  the  first  test  session.  This  participant 
had  prior  experience  with  doing  the  Stroop  test,  which  may  have  influenced  this  result.  The 
workload  level  did  not  change  as  mnch  as  engagement,  but  we  noticed  that  native  English 
speakers  (S004,  S005,  S007)  had  higher  average  workload  when  doing  tests.  S007  showed  a 
somewhat  different  trend  from  other  participants  in  both  engagement  and  workload.  Based  on 
the  survey  result,  she/  he  did  not  have  enough  sleep  in  the  previous  night  (about  5  hours)  and 
did  not  feel  stressed  when  the  timer  has  started. 

Figure  8  shows  GSR  of  all  participants.  The  range  of  skin  resistance  is  individual  dependent, 
so  every  participant  has  her/  his  own  baseline.  The  unit  of  this  figure  is  /xS;  a  higher  conductance 
indicates  a  higher  arousal.  In  the  experiment,  the  arousal  level  was  associated  with  the  stress 
level.  Thus,  the  curves  were  fitted  for  the  difficulty  level,  and  were  similar  to  the  change  in 
engagement. 

In  the  interest  of  how  biosignals  change  with  the  outcome  of  each  question,  signals  measured 
from  the  EEG  headset,  the  GSR  sensor,  the  eye  tracker,  and  the  time  spent  on  each  question 
were  stacked  up  with  the  mark  of  questions  answered  incorrectly.  For  example.  Figure  9  is  the 
plot  of  S005.  She/  he  showed  very  low  engagement/  very  high  distraction  in  some  questions 
in  the  last  session,  where  many  errors  occurred  (9/20).  Further  experiments  and  analyses  are 
needed  for  identifying  the  causality.  In  addition,  when  she/  he  noticed  an  error  was  made,  or 
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Figure  6:  The  overview  of  the  engagement  level  of  all  participants  through  the  test.  Engagement 
is  a  scale-normalized  metric. 


Figure  7:  The  overview  of  the  workload  level  of  all  participants  through  the  test.  Workload  is 
a  scale-normalized  metric. 
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Figure  8:  The  overview  of  the  mental  stress  level  of  all  participants  through  the  test.  The  range 
or  skin  resistance  varies  with  people. 


adjusted  her/  his  state  from  rest  to  answer  questions,  the  stress  level  increased.  One  of  the 
most  common  reasons  for  incorrect  responses  was  to  run  out  of  time,  participants  tended  to 
answer  in  a  very  short  period  for  the  question  that  followed  a  wrongly  answered  one. 

In  the  next  section,  we  are  going  to  further  look  into  the  9  raw  brain  waves.  More  details 
about  what  we  could  understand  from  EEG  will  be  discussed. 


5  Approach  to  Raw  Data  Analysis 

In  addition  to  analyzing  the  GSR,  eye  movement  and  EEG  metrics,  two  methods  traditionally 
applied  to  time  series  data  of  dynamic  systems  were  applied  to  the  EEG  data  sets.  Eirst 
principal  component  analysis  (PGA)  was  applied  to  identify  differences  across  data  sets  acquired 
under  different  testing  conditions.  Independent  component  analysis  (IGA)  is  often  used  in  EEG 
analysis  instead  of  PGA  because  of  its  ability  to  detect  and  remove  artifacts  in  the  data  such  as 
eye-blinks  [8] .  IGA  has  also  been  shown  able  to  correlate  brain  activity  to  behavioral  phenomena 
including  workload  [14]  and  drowsiness  [16].  PGA  is  chosen  here  in  order  to  investigate  its  ability 
to  distinguish  relatively  long  time  histories  acquired  under  different  test  conditions.  Purther, 
training  data  is  not  necessary  to  perform  the  PGA  analysis  presented  here. 

The  second  method  is  the  least  squares  complex  exponential  (LSGE)  parameter  estimation 
technique.  The  purpose  of  applying  this  technique  to  the  EEG  data  is  to  investigate  the  ability 
of  the  LSGE  algorithm  to  identify  a  model  for  the  data  and  then  to  use  that  model  to  make  a 
one-step-ahead  prediction  of  the  data.  Both  the  PGA  and  the  LSGE  analysis  are  described  in 
the  following  sections. 
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Figure  9:  EEG,  GSR,  and  pupil  dilation  measured  from  one  participant  (S005)  while  doing  the 
test.  The  last  raw  shows  the  time  spent  on  each  question  and  rest  sessions. 


5.1  Principal  Component  Analysis 

Principal  component  analysis  (PGA)  uses  singular  value  decomposition  to  identify  principal 
values  and  vectors  of  a  data  set.  The  magnitude  of  the  principal  values  indicate  the  contribution 
of  the  corresponding  vector  in  describing  the  data.  In  this  work,  PGA  was  applied  to  the  set 
of  EEG  time  histories  acquired  from  the  bAlert  system.  Data  from  the  nine  data  channels 
{F3,Fz,Fi,C3,Cz,Ci,Pi,POz,P4:)  shown  in  Eigure  2  were  compiled  into  a  single  matrix,  Q,  as 


follows 

'Fsih) 

Fzih) 

Fi{ti) 

Csih) 

Cz{ti) 

C4{tl) 

mi) 

POzih) 

Piih)' 

^3(^2) 

Fzih) 

Fi{t2) 

Csih) 

Cziti) 

C4ih) 

pm) 

POz{t2) 

pm) 

Q  = 

_F3{tj\f) 

Fz{tN) 

F4{tN) 

Cz{tN) 

CiitN) 

Psitj^) 

POzit^) 

P4{tN)_ 

(1) 


where  N  is  the  number  of  data  points  in  the  selected  data  set.  Singular  value  decomposition 
was  then  applied  to  matrix  Q  to  identify  the  left  (vl)  and  right(uij)  singular  (principal)  vectors 
and  the  singular  (principal)  values  (SV): 

[uL,5'V,Ufl]  =  svd((3).  (2) 

The  singular  values  were  used  to  identify  which  vectors  contribute  most  to  the  overall 
measured  response.  The  corresponding  vectors  were  then  studied  to  identify  the  data  channels 
which  were  most  affected  by  the  changing  conditions  of  the  test.  This  study  focused  on  the 
effect  of  the  presence  of  a  timer  while  subjects  answered  questions,  as  described  in  Section  3. 
Therefore,  the  data  from  a  chosen  subject  was  partitioned  into  three  sets.  The  first  set  contained 
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the  data  from  the  first  forty  questions,  in  which  no  timer  was  present.  The  second  set  contained 
the  data  from  the  second  forty  questions,  in  which  a  timer  of  3  seconds  was  imposed  on  the 
subject.  Finally,  the  third  set  contained  data  from  the  last  forty  questions,  in  which  a  timer 
of  1.5  seconds  was  imposed.  The  hypothesis  explored  with  this  analysis  is  that  changes  in 
the  singular  values  and  vectors  from  the  different  data  sets  correspond  to  changes  in  the  brain 
function  caused  by  the  change  in  test  conditions. 


5.2  Least  Squares  Complex  Exponential  Method 

In  dynamic  systems  analysis,  the  least  squares  complex  exponential  (LSCE)  method  [2]  is  used 
to  identify  modal  parameters  from  time  histories  acquired  from  multiple  input-output  pairs. 
A  parallel  exists  between  dynamic  data  acquired  using  a  single  excitation  source  and  multiple 
sensors  to  measure  a  system’s  response  and  the  EEG  data  acquired  here.  In  both  cases,  the 
system  is  excited  by  a  source.  For  traditional  dynamic  systems,  the  source  is  typically  an 
applied  force  to  the  system.  In  the  experiment  presented  here,  a  subject’s  brain  is  excited 
by  the  presentation  of  a  question.  In  traditional  dynamics  applications,  the  system’s  response 
will  be  affected  by  its  physical  properties,  boundary  conditions,  and  environmental  factors. 
Similarly,  the  response  of  the  subject’s  brain  will  be  affected  by  his  or  her  current  mental  state 
(levels  of  fatigue,  engagement,  etc.),  the  difficulty  of  the  task,  and  environmental  conditions. 
With  these  parallels  in  mind,  the  LSCE  method  was  applied  to  the  EEG  data. 

The  purpose  of  the  LSCE  method  is  to  identify  the  parameters,  a^,  such  that 


^  ^  ^k^n(ii+k)  —  0 
fc=0 


(3) 


where  m  is  the  model  order,  hn{t)  is  an  impulse  response  function,  n  indicates  the  data  channel 
of  interest,  and  i  is  an  arbitrary  starting  index  within  the  time  vector,  t.  Note  that  the  index 
i  must  be  selected  such  that  z  -I-  m  is  within  the  bounds  of  the  data.  Because  an  impulse 
response  function  could  not  be  directly  determined  from  the  data  acquired  from  the  EEG  sensors 
(impulse  response  function  calculations  require  the  input  force  to  be  measured) ,  an  operational 
impulse  response  function  was  used  instead.  As  detailed  in  [3] ,  the  operational  impulse  response 
function  of  a  time  history,  x(t),  is  the  positive  lags  of  the  autocorrelation  function,  R^x-  The 
operational  impulse  response  function  will  be  denoted  as  h{t),  with  the  "  distinguishing  it  from 
the  standard  impulse  response  function.  After  substituting  in  the  operational  impulse  response 
function,  normalizing  the  terms  by  and  rearranging.  Equation  3  becomes 


<^ohn{ti+o)  +  Oilhn{ti+l)  +  ■  ■  ■  +  a  m—l^n(j‘i+m  —  l')  ^n(fz+m) 


(4) 


Equation  4  has  m  —  1  unknowns,  and,  therefore,  more  equations  are  necessary  in  order  to 
determine  the  values  for  the  a^’s.  The  additional  equations  come  from  the  N  different  data 
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channels  as  well  as  altering  the  starting  index  i  within  the  same  data  stream: 


hi  (to) 
hi{ti) 

hi(ti)  .. 

hi(t2) 

■  •  hi  (^tfji—l) 

■  •  hi  {trn) 

hi  {tm) 
hi  (tm+l) 

hi  (im—l) 

h2(to) 

h2(ti) 

hi  (fm)  •  ■ 

4(^2)  •  . 

hl{t2m-2) 

h2{tm-l) 

■  •  h2(tm) 

ao 

ai 

hl{t2m+l) 
h2  {tm) 
h2{tm+l) 

h2(tm-l) 

h2  (fm)  •  ■ 

h2{t2m-2) 

J^m—l_ 

h2{t2m+l) 

hjv(to) 

h]v(ii) 

hN{tl) 

hN{t2) 

hN{tm-l) 

■  •  h]\[  {trn) 

hj\[{tm) 

hN{tm+l) 

_hN(tm-l) 

hN  (fm)  •  ■ 

hN{t2m-2)_ 

hN{t2m+l)_ 

After  solving  Equation  5,  the  and  /i„  values  were  substituted  into  Equation  3  and  the 
mean  and  standard  deviation  of  the  model  error  were  calculated.  The  results  of  these  analyses 
will  be  presented  in  Section  6. 


6  Results 

6.1  Principal  Component  Analysis 

To  explore  the  potential  of  using  PCA  to  assess  brain  function,  data  from  one  subject  (S005) 
was  analyzed  in  depth.  Future  work  will  expand  this  approach  to  include  data  from  multiple 
subjects. 

Figure  10  shows  the  singular  values  for  the  data  from  the  three  test  sections  (i.e.  no  timer, 
3  second  timer,  1.5  second  timer)  acquired  from  the  chosen  subject.  In  general,  the  trends  for 
all  three  data  sets  is  similar.  For  all  three  data  sets,  the  first  singular  value  is  significantly 
larger  than  the  other  eight  indicating  that  the  first  component  contributes  most  strongly  to 
the  overall  response.  The  singular  values  of  components  two  through  five  form  a  cluster  of 
components  that  contribute  less  than  the  first,  but  are  distinct  from  components  six  through 
nine.  The  dropoff  in  magnitude  of  the  singular  values  after  component  five  indicates  that  the 
contributions  of  components  six  through  nine  are  not  as  significant  as  those  of  the  first  five. 
Therefore,  only  the  first  five  components  will  be  used  in  subsequent  analyses. 

Figure  11  shows  the  amplitudes  of  the  first  right  singular  vectors  for  the  three  data  sets. 
Singular  vectors  determined  by  singular  value  decomposition  form  an  orthogonal  set  of  vectors 
and  are  analogous  to  operational  mode  shapes  in  traditional  vibrations  analysis.  The  singular 
vectors  indicate  the  relative  amplitude  of  response  of  each  of  the  electrodes  for  the  selected 
component.  Changes  in  the  amplitude,  or  ’’shape”,  of  the  vector  indicates  a  change  in  the 
measured  response  for  that  data  set.  In  Figure  11,  the  three  vectors  have  distinct  shapes.  A 
possible  explanation  for  these  differences  is  that  the  addition  of  a  timer  affected  the  subject’s 
brain  function  due  to  perceived  stress  caused  by  the  time  pressure. 

Figure  12  shows  the  right  singular  vectors  for  components  two  through  five  for  the  three  data 
sets.  Component  two  (vfo)  is  fairly  consistent  across  data  sets,  but  the  other  three  components 
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Electrode 


Figure  11:  First  right  singular  vectors  for  the  data  acquired  with  no  timer  (blue),  3  second 
timer  (green)  and  1.5  second  timer  (red). 


exhibit  distinct  differences.  In  general,  the  trends  exhibited  by  the  vectors  associated  with  the 
other  three  components  (vjjs,  vm,  vn^)  progress  from  the  un-timed  data  set  (blue),  to  the  data 
set  with  a  3  second  time  (green),  to  the  data  set  with  the  1.5  second  timer.  For  example,  the 
plot  of  the  VRi  vectors  shows  that  the  shape  flattens  out  near  the  C3  node  as  the  time-to- 
answer  is  decreased.  Conversely,  the  shape  peaks  more  near  the  (74  node  as  the  time-to-answer 
decreases.  Although  these  trends  are  for  a  single  subject,  they  show  the  potential  for  this  type 
of  analysis  to  distinguish  mental  states.  Future  work  will  be  needed  to  establish  statistically 
significant  correlations  between  the  trends  observed  here  and  the  human  state. 

6.2  Least  Squares  Complex  Exponential  Analysis 

A  long  term  goal  of  this  work  is  develop  predictive  capabilities  that  include  determining  when 
a  person  is  susceptible  to  poor  decision  making.  Model  development  is  a  key  component  of 
that  effort.  To  determine  the  effectiveness  of  the  model  presented  in  Equation  3,  Equation  5 
was  solved  using  data  from  the  nine  electrode  channels  and  a  model  order  of  16.  Data  was 
first  partitioned  by  question,  and  a  model  was  determined  for  the  data  from  each  question 
individually.  This  partitioning  allowed  for  comparisons  to  be  made  across  testing  conditions, 
since  the  time-to-answer  was  decreased  for  the  three  sets  of  questions.  To  determine  how  well 
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Figure  12:  Second  through  fourth  right  singular  vectors  for  the  data  acquired  with  no  timer 
(blue),  3  second  timer  (green)  and  1.5  second  timer  (red). 


the  model  describes  the  data,  the  distributions  of  the  mean  and  standard  deviation  of  the 
percent  error  for  each  question  were  determined  and  are  shown  in  Figure  13.  These  results 
show  that  the  model  is  most  accurate  when  the  time  pressure  is  greatest.  Figure  13  A  shows 
that  for  26  of  the  40  questions  presented  with  the  shortest  timer,  the  model  error  is  less  than 
12%  error.  Eighty-five  percent  of  all  of  the  data  was  modeled  with  less  than  25%  error.  This 
model  represents  one-step-ahead  predictive  capability.  Ongoing  work  is  being  conducted  to 
improve  both  its  accuracy  and  predictive  capability. 


7  Conclusions 

In  this  paper  we  describe  a  dynamic  data  driven  approach  to  predict  operator  error  taking 
advantage  of  the  advances  in  commercial  grade  neuro-physiological  sensors.  Some  elements  of 
our  overall  approach  described  in  this  paper  are:  (1)  a  human  system  DDDAS  is  implemented 
as  a  classical  dynamic  data  driven  application  system.  The  human  operator  is  wired  with 
commercial  grade  B-alert,  shimmer,  and  eye-tracking  sensors.  Data  from  these  sensors  are 
assimilated  in  an  android  body  area  network  and  stream  into  the  stress-intensity  simulation 
model.  The  simulation  model  assesses  the  stress  and  intensity  of  stress  in  the  operator  and 
generates  one  period  prediction;  (2)  an  experimental  method  to  collect  data  on  human  subject; 
and  (3)  analysis  and  prediction  approach  based  on  principal  component  analysis  with  singular 
value  decomposition.  The  preliminary  result  of  this  study  is  promising.  We  have  shown  that 
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(A) 


(B) 


Figure  13:  Distributions  of  (A)  mean  percent  error  and  (B)  standard  deviation  of  percent  error 
for  the  LSCE  model  calculated  using  data  acquired  with  no  timer  (blue),  3  second  timer  (green) 
and  1.5  second  timer  (red). 


under  stressful  conditions,  the  model  can,  with  reasonable  certainty,  predict  the  occurrence  of 
an  error  one  period  in  advance.  This  is  significant  and  with  improvement  of  prediction  and 
sensor  fusion  algorithms,  we  hope  to  make  predictions  of  errors  earlier. 
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